Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset statistics #5768

Merged
merged 68 commits into from
Jan 10, 2025
Merged

Dataset statistics #5768

merged 68 commits into from
Jan 10, 2025

Conversation

YohannParis
Copy link
Member

@YohannParis YohannParis commented Dec 5, 2024 •
edited
Loading

Description

  • Add a pandas stat task from a Dataset
  • Updated DatasetColumn with a new property stats with proper object representation
  • Add a new Service DatasetStatistics to handle the task and update the dataset accordingly
  • Call the new service in two places, on upload of Dataset from the front-end and when adding a new file

@YohannParis YohannParis requested a review from kbirk December 5, 2024 22:00
@YohannParis YohannParis linked an issue Dec 5, 2024 that may be closed by this pull request
[TASK]: Get Dataset stats #5754
Closed
YohannParis and others added 26 commits December 5, 2024 16:12
@YohannParis
Merge branch 'main' into dataset-statistics
000a912
@YohannParis
Merge branch 'main' into dataset-statistics
315ef25
@YohannParis
Merge branch 'main' into dataset-statistics
8397e13
@YohannParis
remove the unnecessary util file
88218d4
@YohannParis
Add messages and python/pandas type to ColumnType
3a67e93
@YohannParis
Merge branch 'main' into dataset-statistics
7785c63
@YohannParis
Create DatasetStatistics.java
72a5622
@YohannParis
Update DatasetStatistics.java
466c018
@YohannParis
Update DatasetService.java
9205194
@YohannParis
change when this is called
225f737
@YohannParis
cleaning up
b2c6efe
@YohannParis
Merge branch 'main' into dataset-statistics
dc68235
@YohannParis
add task notifications
8e32a59
@github-actions
chore: lint types
0424577
@YohannParis
Setup proper types for the column statistics
88d91b1
@YohannParis
Merge branch 'main' into dataset-statistics
9ca632d
@YohannParis
add the presignedURL
77e9de7
@YohannParis
Update DatasetStatistics.java
2774c28
@YohannParis
Do it properly
e583056
@YohannParis
Merge branch 'main' into dataset-statistics
e9e1106
@YohannParis
use the preseigned URL to pass to the task
ddf812c
@YohannParis
Check that the response might be null
61c6407
@YohannParis
better error checking
b5c701a
@YohannParis
improve readibility and format response properly
8133bac
@YohannParis
Update DatasetStatistics.java
384217f
@YohannParis
fix the issue with 🐍 to 🐫 case
37fa82e
@YohannParis YohannParis marked this pull request as ready for review January 9, 2025 21:55
@YohannParis YohannParis merged commit 89b4b99 into main Jan 10, 2025
6 of 7 checks passed
@YohannParis YohannParis deleted the dataset-statistics branch January 10, 2025 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mloppie mloppie mloppie approved these changes

@dvince2 dvince2 dvince2 approved these changes

@mwdchang mwdchang mwdchang approved these changes

@kbirk kbirk Awaiting requested review from kbirk

@dgauldie dgauldie Awaiting requested review from dgauldie dgauldie is a code owner

@jryu01 jryu01 Awaiting requested review from jryu01 jryu01 is a code owner

@Tom-Szendrey Tom-Szendrey Awaiting requested review from Tom-Szendrey Tom-Szendrey is a code owner

@shawnyama shawnyama Awaiting requested review from shawnyama shawnyama is a code owner

@blanchco blanchco Awaiting requested review from blanchco blanchco is a code owner

@asylves1 asylves1 Awaiting requested review from asylves1 asylves1 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[TASK]: Get Dataset stats
5 participants
@YohannParis @mwdchang @dvince2 @mloppie @dgauldie